Big data is a term for data sets that are so large or complex that traditional data processing application software is inadequate to deal with them. Big data challenges include capturing data, data storage, data analysis, search, sharing, transfer, visualization, querying, updating and information privacy.
The data could be from social networks, web server logs, traffic flow sensors, satellite imagery, broadcast audio streams, banking transactions, MP3s of rock music, the content of web pages, scans of government documents, GPS trails, telemetry from automobiles, financial market data and so on.
Potential Uses of Big Data:
(i) Seed Selection – Big-data businesses can analyse varieties of seeds across numerous fields, soil types, and climates and select the best.
(ii) Irrigation – Precision agriculture aids farmers in tailored and effective water management, helping in production, improving economic efficiency and minimising waste and environmental impact.
(iii) Food processing – They help in streamlining food processing value chains by finding the core determinants of process performance, and acting to continually improve the accuracy, quality and yield of production. They also optimise production schedules based on supplier, customer, machine availability and cost constraints.
(iv) Crop disease – Similar to the way in which Google can identify flu outbreaks based on where web searches are originating, analysing crops across farms helps identify diseases that could ruin a potential harvest.
(v) Weather – Advanced analytics capabilities and agri-robotics such as aerial imagery, sensors help provide sophisticated local weather forecasts can help increasing global agricultural productivity over the next few decades.
(vi) Climate Change – Since, climate change and extreme weather events will demand proactive measures to adapt or develop resiliency, Big Data can bring in the right information to take informed decisions.
(vii) Loss control – In India, every year 21 million tons of wheat is lost, primarily due to scare cold-storage centres and refrigerated vehicles, poor transportation facilities and unreliable electricity supply. Big Data has the potential of systematisation of demand forecasting thus reducing such losses.
(viii) Pricing – A trading platform for agricultural commodities that links small-scale producers to retailers and bulk purchasers via mobile phone messaging can help send up-to-date market prices via an app or SMS and connect farmers with buyers, offering collective bargaining opportunities for small and marginal farmers.
(ix) Consumer habits can be studied, and policies can be made which will be in line with the demands of consumers.
(x) Some other benefits – Solving traffic problems in cities, targeting healthcare delivery, efficient supply chain management, providing a personalized educational experience for students, enabling security to individuals and society at large.
-> Any breach of confidentiality regarding data that is collected and processed by the government could have serious ramifications.
-> Sharing data between departments and across ministries is a challenge, given the jurisdictional boundaries that exist.
-> Authenticity of data is another challenge.
A change in mindset and effective training is required to make data-driven decisions. A comprehensive Big Data programme across Central and state government ministries/departments with help from industry, academic and research institutions should be formulated.