-
Notifications
You must be signed in to change notification settings - Fork 0
/
working_with_json_response.balnotebook
1 lines (1 loc) · 18.7 KB
/
working_with_json_response.balnotebook
1
[{"kind":1,"language":"markdown","value":"In this sample, we will be looking at some key features of Ballerina using the HTTP client to retrieve population data via the [World Bank Indicators API](https://datahelpdesk.worldbank.org/knowledgebase/articles/889392-about-the-indicators-api-documentation) and then processing the retrieved data.","outputs":[],"executionSummary":{},"metadata":{}},{"kind":1,"language":"markdown","value":"This sample demonstrates the following.\n- Using an HTTP client to retrieve JSON data\n- Working directly with the JSON payload\n- Query expressions\n- Defining and using application-specific types corresponding to JSON payload","outputs":[],"executionSummary":{},"metadata":{}},{"kind":1,"language":"markdown","value":"Let's first import the required modules. ","outputs":[],"executionSummary":{},"metadata":{}},{"kind":2,"language":"ballerina","value":"import ballerina/http;\nimport ballerina/io;","outputs":[],"executionSummary":{},"metadata":{}},{"kind":1,"language":"markdown","value":"Let's now implement the logic step by step.","outputs":[],"executionSummary":{},"metadata":{}},{"kind":1,"language":"markdown","value":"Let's initialize an `http:Client` object specifying the URL for the World Bank API. Note that we have had to pass 1.1 as the HTTP version, since the Ballerina HTTP client defaults to 2.0 as the version, but the backend doesn't support the same.","outputs":[],"executionSummary":{},"metadata":{}},{"kind":2,"language":"ballerina","value":"final http:Client worldBankClient = check new (\"http://api.worldbank.org/v2\", httpVersion = http:HTTP_1_1);","outputs":[],"executionSummary":{},"metadata":{}},{"kind":1,"language":"markdown","value":"Let's assume we have a parameter named `country` that holds the country code for the data we are interested in. This can also be a variable (local, configurable, module-level, etc.) or even a constant.","outputs":[],"executionSummary":{},"metadata":{}},{"kind":1,"language":"markdown","value":"Let's retrieve the data for the country in JSON format and write it to a file to examine the data.","outputs":[],"executionSummary":{},"metadata":{}},{"kind":2,"language":"ballerina","value":"function retrieveData(string country) returns error? {\n json payload = check worldBankClient->get(string `/country/${country}/indicator/SP.POP.TOTL?format=json`);\n check io:fileWriteJson(\"all.json\", payload);\n}","outputs":[],"executionSummary":{},"metadata":{}},{"kind":1,"language":"markdown","value":"Notes:\n- `country` is used as an interpolation in the string template expression that is the argument to `get`. See [string template expressions](https://ballerina.io/learn/by-example/backtick-templates/).\n- the `get` remote method uses the contextually-expected type (`json` from the left-hand side here) to try and bind the retrieved payload to the specific type. If the attempt to convert/parse as the specific type fails, an error will be returned. See [API docs for the HTTP client's `get` method](https://lib.ballerina.io/ballerina/http/latest/clients/Client#get) and [dependently-typed functions](https://ballerina.io/learn/by-example/dependent-types/).\n- `remote` and `resource` methods indicate network interactions. Such methods have to be called using the `->` syntax. This differentiates between network calls and normal function/method calls.\n- The `get` method and the `io:fileWriteJson` function may return an error value at runtime. Using `check` with an expression that may evaluate to an error results in the error being returned immediately if the expression evaluates to an error at runtime. See [`check` expression](https://ballerina.io/learn/by-example/check-expression/). ","outputs":[],"executionSummary":{},"metadata":{}},{"kind":2,"language":"ballerina","value":"public function main() returns error? {\n check retrieveData(\"LK\");\n}","outputs":[],"executionSummary":{},"metadata":{}},{"kind":1,"language":"markdown","value":"Examining the content written to the file, we can observe the following.\n- the JSON payload is a JSON array of two items\n- the first item is a JSON object with information about the data (e.g., pagination, last updated, etc.)\n- the second item is another array of JSON objects where each object contains population data for a particular year","outputs":[],"executionSummary":{},"metadata":{}},{"kind":1,"language":"markdown","value":"### Working directly with JSON\n\nNow that we know the structure of the payload, let's update the `retrieveData` function to do the following.\n- retrieve the payload as a JSON array by changing the expected type (type of `payload`) to `json[]`.\n- once we have the array, access the second member of the array (population data by year) and ensure its type is `json[]` (in line with what we observed when examining the payload)\n- then iterate through that array and collect population data at the end of each decade","outputs":[],"executionSummary":{},"metadata":{}},{"kind":2,"language":"ballerina","value":"function retrieveData(string country) returns error? {\n json[] payload = check worldBankClient->get(string `/country/${country}/indicator/SP.POP.TOTL?format=json`);\n \n json[] populationByYear = check payload[1].ensureType();\n\n json[] populationEveryDecade = from json population in populationByYear\n let string yearStr = check population.date,\n int year = check int:fromString(yearStr)\n where year % 10 == 0\n select population;\n \n check io:fileWriteJson(\"population_every_decade.json\", populationEveryDecade);\n}","outputs":[],"executionSummary":{},"metadata":{}},{"kind":1,"language":"markdown","value":"Notes:\n- [`value:ensureType`](https://lib.ballerina.io/ballerina/lang.value/0.0.0/functions#ensureType) works similar to a cast, but returns an error instead of panicking if the value does not belong to the target type. It is also a dependently-typed function and infers the `typedesc` to ensure against from the expected type (`json[]` here) if not explicitly specified.\n- when `check` is used with JSON access and the expected type is a subtype of `()|boolean|int|float|decimal|string`, it is equivalent to using `value:ensureType` with the JSON access. For example, `string yearStr = check population.date` is equivalent to `string yearStr = check value:ensureType(population.date)`.\n\n See\n - https://ballerina.io/learn/by-example/access-json-elements\n - https://medium.com/ballerina-techblog/ballerinas-json-type-and-lax-static-typing-3b952f6add01 \n - https://medium.com/ballerina-techblog/ballerina-working-with-json-part-i-json-to-record-conversion-1e810b0a30f0 \n \n- a [query expression](https://ballerina.io/learn/by-example/#query-expressions) is used to create a JSON array with just the information from the years (end of a decade) we are interested in\n - a [`let` clause](https://ballerina.io/learn/by-example/let-clause/) in a query expression allows declaring temporary variables that will be visible in the rest of the query expression\n - a `where` clause can be used to filter based on the (boolean) result of an expression\n - a `select` clause specifies the value to include. In this example we select the entire population JSON object as is.","outputs":[],"executionSummary":{},"metadata":{}},{"kind":1,"language":"markdown","value":"Examining the data written to `population_every_decade.json` now, we can see that it consists of only the filtered data.","outputs":[],"executionSummary":{},"metadata":{}},{"kind":1,"language":"markdown","value":"Let's now extract out just the population against the year at the end of each decade. Let's use a query expression to add this information to an in-memory map instead of writing it to a file.","outputs":[],"executionSummary":{},"metadata":{}},{"kind":2,"language":"ballerina","value":"function retrieveData(string country) returns map<int>|error {\n json[] payload = check worldBankClient->get(string `/country/${country}/indicator/SP.POP.TOTL?format=json`);\n \n json[] populationByYear = check payload[1].ensureType();\n\n return map from json population in populationByYear\n let string yearStr = check population.date,\n int year = check int:fromString(yearStr)\n where year % 10 == 0\n select [yearStr, check population.value];\n}","outputs":[],"executionSummary":{},"metadata":{}},{"kind":1,"language":"markdown","value":"Notes:\n- the return type of the function has been changed to to allow returning a map of integers now\n- in order to create a map with a query expression, the `map` keyword needs to be used before the `from` keyword. The first expression in the list constructor in the `select` clause is used as the key and the second expression is used as the value. Also see [Creating maps with query expressions](https://ballerina.io/learn/by-example/create-maps-with-query/).\n- the compiler uses `int` (from the return type) as the expected type for `check population.value`, which allows us to use the previously discussed convenient way of accessing JSON members and asserting the type","outputs":[],"executionSummary":{},"metadata":{}},{"kind":1,"language":"markdown","value":"Printing the result returned from this function call, we can now examine the data in the map.","outputs":[],"executionSummary":{},"metadata":{}},{"kind":2,"language":"ballerina","value":"public function main() returns error? {\n map<int> populationByDecade = check retrieveData(\"LK\");\n io:println(populationByDecade);\n}","outputs":[],"executionSummary":{},"metadata":{}},{"kind":1,"language":"markdown","value":"Output would look similar to\n\n```cmd\n{\"2020\":21919000,\"2010\":20261738,\"2000\":18777606,\"1990\":17325769,\"1980\":15035840}\n```","outputs":[],"executionSummary":{},"metadata":{}},{"kind":1,"language":"markdown","value":"The [sequence diagram](https://ballerina.io/why-ballerina/graphical/) generated for this function is the following.\n\n","outputs":[],"executionSummary":{},"metadata":{}},{"kind":1,"language":"markdown","value":"### Working with user-defined types\n\nIn the previous section, we looked at how we can extract the JSON payload and work directly with the JSON values.\n\nAlternatively, we can also convert the payload to specific user-defined types and work with them instead.\n\nTo recap, the payload we received was a list of two members, where the first member was a JSON object with information about the data and the second member was another array of JSON objects containing population data for each year.\n\nWe can model user-defined types for this as follows.\n- since the members of the entire payload JSON array are of two different structures (JSON object and array of JSON objects), we can use a [tuple](https://ballerina.io/learn/by-example/tuples/) to define this structure. Let's call it `PopulationIndicator`. Also see [lists in Ballerina](https://ballerina.io/learn/by-example/#lists).\n- since the first member (information) is a JSON object, we can define a [record](https://ballerina.io/learn/by-example/records/) to represent the structure. Let's call it `IndicatorInfo`. Also see [mappings](https://ballerina.io/learn/by-example/#mappings) and [records](https://ballerina.io/learn/by-example/#records).\n- similarly, we can define a record to represent each JSON object that contains population information. Let's call it `PopulationByYear`. Since the second member of the payload JSON array is a list of these JSON objects (same type), we can use an array of this record type (`PopulationByYear[]`) as the second member of the tuple.","outputs":[],"executionSummary":{},"metadata":{}},{"kind":1,"language":"markdown","value":"Let's first define the `IndicatorInfo` and `PopulationByYear` records. \n\nNote how we are using the exact expected types as the types of the fields in the record (as opposed to `json`). The field names have to be an exact match with those expected in the payload. A [quoted identifier](https://ballerina.io/learn/by-example/identifiers/) (`'decimal`) is used to use a reserved keyword (`decimal`) as the name of a field.","outputs":[],"executionSummary":{},"metadata":{}},{"kind":2,"language":"ballerina","value":"type IndicatorInfo record {|\n int page;\n int pages;\n int per_page;\n int total;\n string sourceid;\n string sourcename;\n string lastupdated;\n|};\n\ntype PopulationByYear record {|\n record {|\n string id;\n string value;\n |} indicator;\n record {|\n string id;\n string value;\n |} country;\n string countryiso3code;\n string date;\n int value;\n string unit;\n string obs_status;\n int 'decimal;\n|};","outputs":[],"executionSummary":{},"metadata":{}},{"kind":1,"language":"markdown","value":"We can now define `PopulationIndicator` using these records.","outputs":[],"executionSummary":{},"metadata":{}},{"kind":2,"language":"ballerina","value":"type PopulationIndicator [IndicatorInfo, PopulationByYear[]];","outputs":[],"executionSummary":{},"metadata":{}},{"kind":1,"language":"markdown","value":"We can now use this type directly when calling the `get` method. The HTTP client will retrieve the JSON payload and attempt the conversion to `PopulationIndicator` itself. In case the conversion fails the `get` method will return an error. ","outputs":[],"executionSummary":{},"metadata":{}},{"kind":2,"language":"ballerina","value":"function retrieveData(string country) returns map<int>|error {\n PopulationIndicator payload = check worldBankClient->get(string `/country/${country}/indicator/SP.POP.TOTL?format=json`);\n \n return map from PopulationByYear population in payload[1]\n let string yearStr = population.date,\n int year = check int:fromString(yearStr)\n where year % 10 == 0\n select [yearStr, population.value];\n}","outputs":[],"executionSummary":{},"metadata":{}},{"kind":1,"language":"markdown","value":"Note how this simplified the rest of the code too.\n- we no longer have to use `value:ensureType` to retrieve the second member as an array, since the conversion is done to an array of `PopulationByYear` already\n- we no longer have to use check when accessing the `date` and `value` fields since the record conversion also handled the type validation (`string` an `int` respectively)\n\n> HTTP data binding uses the [`value:cloneWithType` lang library function](https://lib.ballerina.io/ballerina/lang.value/0.0.0/functions#cloneWithType) internally, which we could also use directly for conversion.","outputs":[],"executionSummary":{},"metadata":{}},{"kind":1,"language":"markdown","value":"In the mapping above we've specified each field explicitly. Alternatively, we could also leverage [open records](https://ballerina.io/learn/by-example/open-records/) and [controlling openness](https://ballerina.io/learn/by-example/controlling-openness/) to explicitly specify only the fields we are interested in.\n\nFor example, we can explicitly specify only the `date` and `value` fields in `PopulationByYear`, since they are the only fields we are intersted in. As for the rest of the fields, we use the `json` type in the record rest descriptor to just say the rest of the fields have to be/are `json` values. Similarly, since we are not interested in the first member of the payload JSON array, we can avoid specifying a separate type for it.","outputs":[],"executionSummary":{},"metadata":{}},{"kind":2,"language":"ballerina","value":"type PopulationByYear record {|\n string date;\n int value;\n json...;\n|};\n\ntype PopulationIndicator [json, PopulationByYear[]];","outputs":[],"executionSummary":{},"metadata":{}},{"kind":1,"language":"markdown","value":"Defining user-defined (application-specific) types to represent JSON payload has numerous benefits, including\n- validating the payload (structure and types) in one go\n- compile-time validation of field/member access\n- better tooling experience (e.g., completion, code actions)\n\nHowever, conversion is a somewhat expensive operation, and if you are not interested in all the data or are interested only in a limited number of members (compared to the total number of members), direct access may be a better approach.","outputs":[],"executionSummary":{},"metadata":{}},{"kind":1,"language":"markdown","value":"#### Generating user-defined types\n\nWhile these records could be defined by manually, you could use the [Paste JSON as Record](https://wso2.com/ballerina/vscode/docs/edit-the-code/commands/) VSCode command to generate the initial records and update/refine if/as necessary. This way we wouldn't have to manually define each field/record.","outputs":[],"executionSummary":{},"metadata":{}},{"kind":1,"language":"markdown","value":"#### Using binding patterns","outputs":[],"executionSummary":{},"metadata":{}},{"kind":1,"language":"markdown","value":"Ballerina suppports binding patterns which allow extracting separate parts of a structured value to separate variables in one go. Binding patterns are quite powerful and can be used in various constructs including variable declarations, `foreach` statements, the `from` clause in query expressions/actions, `match` statements, etc.\n\nSee the [examples on binding patterns](https://ballerina.io/learn/by-example/#binding-patterns) for more details.","outputs":[],"executionSummary":{},"metadata":{}},{"kind":1,"language":"markdown","value":"In the query expression in the `retrieveData` function, we only need to access the `date` and `value` fields from each `PopulationByYear` record. We can use a mapping binding pattern with just those fields to extract and assign them to two variables in the `from` clause itself.","outputs":[],"executionSummary":{},"metadata":{}},{"kind":2,"language":"ballerina","value":"function retrieveData(string country) returns map<int>|error {\n PopulationIndicator payload = check worldBankClient->get(string `/country/${country}/indicator/SP.POP.TOTL?format=json`);\n \n return map from PopulationByYear {date: yearStr, value} in payload[1]\n let int year = check int:fromString(yearStr)\n where year % 10 == 0\n select [yearStr, value];\n}","outputs":[],"executionSummary":{},"metadata":{}},{"kind":1,"language":"markdown","value":"Notes:\n- `date: yearStr` here results in the value of the `date` field being assigned to a variable name `yearStr`. \n- Having just `value` is equivalent to `value: value` in the binding pattern.\n- The types of the newly created variables are decided based on the `PopulationByYear` record here. Alternatively, if `var` is used, the types are inferred from the value.\n- The member binding patterns can also be other structured binding patterns.\n\n ```ballerina\n var {country: {id, value}} = populationByYear\n ```","outputs":[],"executionSummary":{},"metadata":{}}]