--- title: Databricks Structured Streaming - Part 2 (Preparing the Data) description: This series on Databricks will guide you through structured streaming. This post explains how to prepare data that you can use to test structured streaming. image: https://blog.coeo.com/hubfs/101219_JW_databricks.png --- [![](https://www.coeo.com/wp-content/themes/coeo/images/logo.svg)](https://blog.coeo.com/) +44 (0)20 3051 3595 | [info@coeo.com](mailto:info@coeo.com) | [Client portal login](https://my.coeo.com) # Databricks Structured Streaming - Part 2 (Preparing the Data) # The Coeo Blog ![Andy Mitchell](https://blog.coeo.com/hubfs/AndyMCircleLow.jpg) Continuing on from our [previous posts](https://blog.coeo.com/topic/azure-databricks) about Databricks, we are now going to look at structured streaming. For this blog post we are going to use some data from the Seattle fire service that is updated every 5 minutes. The source data we are going to use can be found at [https://data.seattle.gov/Public-Safety/Seattle-Real-Time-Fire-911-Calls/upug-ckch](https://data.seattle.gov/Public-Safety/Seattle-Real-Time-Fire-911-Calls/upug-ckch) 1. Navigate to the link above and familiarise yourself with the type of data that we will be using ![L21-01-01-SeattleFire](https://blog.coeo.com/hs-fs/hubfs/L21-01-01-SeattleFire.png?width=1269&name=L21-01-01-SeattleFire.png)       2. Click on the SOURCE DATASET "Seattle Real Time Fire 911 Calls" to display information about the dataset ![L21-01-02-SeattleFire_about](https://blog.coeo.com/hs-fs/hubfs/L21-01-02-SeattleFire_about.png?width=1277&name=L21-01-02-SeattleFire_about.png)      3. Navigate back to the previous page and click on the "API" link ![L21-01-04-SeattleFire_API](https://blog.coeo.com/hs-fs/hubfs/L21-01-04-SeattleFire_API.png?width=1268&name=L21-01-04-SeattleFire_API.png)      4. Make a note of the URL in the link (copy to the clipboard) ![L21-01-05-SeattleFire_APILink](https://blog.coeo.com/hs-fs/hubfs/L21-01-05-SeattleFire_APILink.png?width=578&name=L21-01-05-SeattleFire_APILink.png)      5. In a new browser window paste in the URL to view the content from the API ![L21-01-05-APIData](https://blog.coeo.com/hs-fs/hubfs/L21-01-05-APIData.png?width=3837&name=L21-01-05-APIData.png)       6. Using the Databricks cluster created in the [previous post](https://blog.coeo.com/databricks-structured-streaming) ![](https://blog.coeo.com/hubfs/image-16.png)      7. Navigate to "Workspace" ![](https://blog.coeo.com/hubfs/image-17.png)      8. Navigate to "Shared" > "Create" > "Folder"  ![](https://blog.coeo.com/hubfs/image-18.png)      9. Enter the name "Introduction to Databricks Structured Streaming"  and click "Create Folder" ![](https://blog.coeo.com/hubfs/image-19.png)       10. Navigate to the folder and click "Create" > "Notebook" ![](https://blog.coeo.com/hubfs/image-20.png)       11. Create a Python notebook called "Part 2 (Preparing the data)" ![](https://blog.coeo.com/hubfs/image-21.png)      12. Enter the following code into the cmd pane of the notebook. These are the variables that we will use for getting the data from the website and saving it to a file. ![](https://blog.coeo.com/hubfs/image-25.png)      13. Get the URL for the data stream and paste it where is above, this can be found from the API button on the following web page [https://data.seattle.gov/Public-Safety/Seattle-Real-Time-Fire-911-Calls/upug-ckch](https://data.seattle.gov/Public-Safety/Seattle-Real-Time-Fire-911-Calls/upug-ckch)       14. Add another cmd pane by clicking on the + that appears below the middle of the previous one and add the following code. This provides a function that we will use save the stream to a file ![](https://blog.coeo.com/hubfs/image.png)      15. Add another cmd pane and add the following code to test that the data is captured to a file ![](https://blog.coeo.com/hubfs/image-24.png)      16. To test the code you can either click the ![](https://blog.coeo.com/hubfs/image-26.png) at the top of the notebook or run each cell individually using the play button in the top right. "Test the function" should return something similar to the following![](https://blog.coeo.com/hubfs/image-27.png)      17. To check that we have some files you can run the above a few times, but leave 5 minutes between executions to allow the source data to refresh.      18. Enter the following in another cmd pane to check that we have some files to process ![](https://blog.coeo.com/hubfs/image-28.png) We now have some data that we can use to test structured streaming. In this post we have: - Created function to pull data from an external API - Created Json files that we can use as the source for our structured streaming dataset. In the next post we will connect to the json files using structured streaming. [![Get more Databricks advice](https://no-cache.hubspot.com/cta/default/3356718/6b83727a-9631-4d72-a88d-eb25b03c82a6.png)](https://cta-redirect.hubspot.com/cta/redirect/3356718/6b83727a-9631-4d72-a88d-eb25b03c82a6) ### Subscribe to Email Updates ## Related posts --- ### [Three ways to profile data with Azure Databricks](https://blog.coeo.com/three-ways-to-profile-data-with-azure-databricks) ### [Azure Data Factory & XML: Loading unsupported file types](https://blog.coeo.com/azure-data-factory-xml-loading-unsupported-file-types) ### [Databricks Structured Streaming - Part 3 (Creating the Stream)](https://blog.coeo.com/databricks-structured-streaming-part3) ### [Databricks Structured Streaming - Part 1 (Creating the Cluster)](https://blog.coeo.com/databricks-structured-streaming) ![](https://www.coeo.com/wp-content/uploads/2016/12/logo-invert.png) +44 (0)20 3051 3595 | info@coeo.com ## Contact Us By clicking submit below, you consent to allow Coeo to store and process the personal information submitted above to provide you the content requested. You may unsubscribe from these communications at any time. For more information on how to unsubscribe and our commitment to your privacy, please review our **[Privacy Policy](https://www.coeo.com/privacy/)**. ## Upcoming Events [See all events](https://www.coeo.com/events/) #### NOW Building, Thames Valley Park Drive, Reading, RG6 1RB [![](https://www.coeo.com/wp-content/themes/coeo/images/social-glass.png)](https://www.glassdoor.co.uk/Overview/Working-at-Coeo-EI_IE959052.11,15.htm)[![](https://www.coeo.com/wp-content/themes/coeo/images/social-in.png)](https://www.linkedin.com/company/coeo-ltd)[![](https://www.coeo.com/wp-content/themes/coeo/images/social-twitter.png)](https://twitter.com/CoeoLtd)[![](https://www.coeo.com/wp-content/themes/coeo/images/social-fb.png)](https://www.facebook.com/coeoltd/) ![](https://www.coeo.com/wp-content/themes/coeo/images//menu-icon.png) ![](https://www.coeo.com/wp-content/themes/coeo/images//menu-close.png) ![](https://www.coeo.com/wp-content/uploads/2016/12/logo-invert.png) +44 (0)20 3051 3595 | [info@coeo.com](mailto:info@coeo.com) - [Solutions](https://www.coeo.com/solutions/) - [Next Steps](https://www.coeo.com/next-steps/) - [Dedicated Support](https://www.coeo.com/dedicated-support/) - [Case studies](https://www.coeo.com/case-studies/) - [Technologies](https://www.coeo.com/solutions/technologies/) - [Industries](https://www.coeo.com/industries/) - [Finance](https://www.coeo.com/industries/finance/) - [Retail](https://www.coeo.com/industries/retail/) - [Technology](https://www.coeo.com/industries/technology/) - [The Team](https://www.coeo.com/people/) - [Join Us](https://www.coeo.com/careers/) - [Graduate Programme](https://www.coeo.com/graduate-programme/) - [About Coeo](https://www.coeo.com/about-coeo/) - [The Coeo Blog](https://www.coeo.com/blog/) - [Contact us](https://www.coeo.com/contact-us/) - [Privacy Notice](https://www.coeo.com/privacy/) - [Cookie Policy](https://www.coeo.com/privacy#Cookie_Policy) - [Events](https://www.coeo.com/events) Sign up to our newsletter ![go arrow](https://www.coeo.com/wp-content/themes/coeo/images/newsletter-go.png) Back to top