Posted by Jennifer Kohl, Google Developers Global Communities Program Manager
Luiza in her hometown, Magas
Magas is the capital of the Republic of Ingushetia, the smallest region in Russia. Centered between Chechnya and North Ossetia, the area is no stranger to conflict. Even as it rebuilds, the region has seen its unemployment numbers rise to as high as 50 percent. Magas, a mostly rural area, is home to a small population of just under six-thousand people - it’s estimated that under 100 of them are developers.
Yet one day that small group of developers decided to take their first step towards becoming a community. These founders heard of Google Developer Groups (GDG) and had seen their community meetups in action on trips to other larger cities in Russia. Inspired by how GDG brought developers together, they believed starting a community in Magas was just what they all needed to grow.
GDG Magas was up and running immediately, hosting small community events in classrooms and meeting spaces across town. And it was there, at a local meetup, where GDG Magas met Luiza.
Luiza speaking at a University competition
At the time, Luiza was a student at a local university. Equipped with a curious mind, she was hungry to learn more. She often challenged herself to think about how women could grow professionally and personally within traditional cultures. Luiza was interested in technology, a mostly unheard of career path in this small town. At the same time, Women Techmakers, a Google program that provides resources for women in technology, started collaborating with GDG chapters around the world to help women like Luiza get started on their journey.
So together, GDG Magas and Women Techmakers started hosting talks and workshops for women in the community. Eventually, they began running a programming class for beginners, and that’s where Luiza realized she had the space to truly explore her interest in code. The community organized thirteen classes, and each Saturday Luiza would join GDG Magas to learn everything from arrays, to Python, to JavaScript, and more.
“I learned everything a beginner needs: numeral systems, loops, algorithms, and even the basics of web development. I was able to work with GDG mentors to improve my skills both in the backend and frontend. Someone was always there to answer my questions.”
With GDG Magas providing Luiza with this access to learning materials and mentorship, there has been no turning back. Luiza landed a competitive role working for an internet company, will soon give her own talks at GDG events, and is even starting her own Google Developer Student Club as she completes her studies in Magas. Luiza is now at the forefront of helping a rural town become a growing tech scene, taking the lead to shape her future and that of many young developers around her.
GDG Magas and similar developer communities are growing faster than ever, thanks to determined developers just like Luiza.
Ready to find a developer community near you? Join a local Google Developer Group, here.
Posted by Wesley Chun, Developer Advocate, Google Cloud
File backup isn't the most exciting topic while analyzing images with AI/ML is more interesting, so combining them probably isn't a workflow you think about often. However, by augmenting the former with the latter, you can build a more useful solution than without. Google provides a diverse array of developer tools you can use to realize this ambition, and in fact, you can craft such a workflow with Google Cloud products alone. More compellingly, the basic principle of mixing-and-matching Google technologies can be applied to many other challenges faced by you, your organization, or your customers.
The sample app presented uses Google Drive and Sheets plus Cloud Storage and Vision to make it happen. The use-case: Google Workspace (formerly G Suite) users who work in industries like architecture or advertising, where multimedia files are constantly generated. Every client job results in yet another Drive subfolder and collection of asset files. Successive projects lead to even more files and folders. At some point, your Drive becomes a "hot mess," making users increasingly inefficient, requiring them to scroll endlessly to find what they're looking for.
A user and their Google Drive files
How can Google Cloud help? Like Drive, Cloud Storage provides file (and generic blob) storage in the cloud. (More on the differences between Drive & Cloud Storage can be found in this video.)
Cloud Storage provides several storage classes depending on how often you expect to access your archived files. The less often files are accessed, the "colder" the storage, and the lower the cost. As users progress from one project to another, they're not as likely to need older Drive folders and those make great candidates to backup to Cloud Storage.
First challenge: determine the security model. When working with Google Cloud APIs, you generally select OAuth client IDs to access data owned by users and service accounts for data owned by applications/projects. The former is typically used with Workspace APIs while the latter is the primary way to access Google Cloud APIs. Since we're using APIs from both product groups, we need to make a decision (for now and change later if desired).
Since the goal is a simple proof-of-concept, user auth suffices. OAuth client IDs are standard for Drive & Sheets API access, and the Vision API only needs API keys so the more-secure OAuth client ID is more than enough. The only IAM permissions to acquire are for the user running the script to get write access to the destination Cloud Storage bucket. Lastly, Workspace APIs don't have their own product client libraries (yet), so the lower-level Google APIs "platform" client libraries serve as a "lowest common denominator" to access all four REST APIs. Those who have written Cloud Storage or Vision code using the Cloud client libraries will see something different.
The prototype is a command-line script. In real life, it would likely be an application in the cloud, executing as a Cloud Function or a Cloud Task running as determined by Cloud Scheduler. In that case, it would use a service account with Workspace domain-wide delegation to act on behalf of an employee to backup their files. See this page in the documentation describing when you'd use this type of delegation and when not to.
Our simple prototype targets individual image files, but you can continue to evolve it to support multiple files, movies, folders, and ZIP archives if desired. Each function calls a different API, creating a "service pipeline" with which to process the images. The first pair of functions are drive_get_file() and gcs_blob_upload(). The former queries for the image on Drive, grabs pertinent metadata (filename, ID, MIMEtype, size), downloads the binary "blob" and returns all of that to the caller. The latter uploads the binary along with relevant metadata to Cloud Storage. The script was written in Python for brevity, but the client libraries support most popular languages. Below is the aforementioned function pseudocode:
drive_get_file()
gcs_blob_upload()
def drive_get_file(fname): rsp = DRIVE.files().list(q="name='%s'" % fname).execute().get['files'][0] fileId, fname, mtype = rsp['id'], rsp['name'], rsp['mimeType'] blob = DRIVE.files().get_blob(fileId).execute() return fname, mtype, rsp['modifiedTime'], blob def gcs_blob_upload(fname, folder, bucket, blob, mimetype): body = {'name': folder+'/'+fname, 'uploadType': 'multipart', 'contentType': mimetype} return GCS.objects().insert(bucket, body, blob).execute()
Next, vision_label_img() passes the binary to the Vision API and formats the results. Finally that information along with the file's archived Cloud Storage location are written as a single row of data in a Google Sheet via sheet_append_row().
vision_label_img()
sheet_append_row()
def vision_label_img(img): body = {'requests': [{'image': {'content': img}, 'features': [{'type': 'LABEL_DETECTION'}]}]} rsp = VISION.images().annotate(body=body).execute().get['responses'][0] return ', '.join('(%.2f%%) %s' % (label['score']*100., label['description']) for label in rsp['labelAnnotations']) def sheet_append_row(sheet_id, row): rsp = SHEETS.spreadsheets().values().append(spreadsheetId=sheet_id, range='Sheet1', body={'values': row}).execute() return rsp.get('updates').get('updatedCells')
Finally, a "main" program that drives the workflow is needed. It comes with a pair of utility functions, _k_ize() to turn file sizes into kilobytes and _linkify() to build a valid Cloud Storage hyperlink as a spreadsheet formula. These are featured here:
_k_ize()
_linkify()
def _k_ize(nbytes): # bytes to KBs (not KiBs) as str return '%6.2fK' % (nbytes/1000.) def _linkify(bucket, fname): # make GCS hyperlink to bucket/folder/file tmpl = '=HYPERLINK("storage.cloud.google.com/{0}/{1}/{2}", "{2}")' return tmpl.format(bucket, folder, fname) def main(fname, bucket, SHEET_ID, folder): fname, mtype, ftime, data = drive_get_img(fname) gcs_blob_upload(fname, folder, bucket, data, mtype) info = vision_label_img(data) sheet_append_row(SHEET_ID, [folder, _linkify(bucket, fname), mtype, ftime, _k_ize(data), info])
While this post may feature just pseudocode, a barebones working version can be accomplished with ~80 lines of actual Python. The rest of the code not shown are constants, error-handling, and other auxiliary support. The application gets kicked off with a call to main() passing in a filename, the Cloud Storage bucket to archive it to, a Drive file ID for the Sheet, and a "folder name," e.g., a directory or ZIP archive. Running it several times results in a spreadsheet that looks like this:
main()
Image archive report in Google Sheets
Developers can build this application step-by-step with our "codelab" - codelabs are free, online, self-paced tutorials - which can be found here. As you journey through this tutorial, its corresponding open source repo features separate folders for each step so you know what state your app should be in after every implemented function. (NOTE: Files are not deleted, so your users have to decide when to their cleanse Drive folders.) For backwards-compatibility, the script is implemented using older Python auth client libraries, but the repo has an "alt" folder featuring alternative versions of the final script that use service accounts, Google Cloud client libraries, and the newer Python auth client libraries.
Finally to save you some clicks, here are links to the API documentation pages for Google Drive, Cloud Storage, Cloud Vision, and Google Sheets. While this sample app deals with a constrained resource issue, we hope it inspires you to consider what's possible with Google developer tools so you can build your own solutions to improve users' lives every day!
Recently, we introduced the "Google Cloud for Student Developers" video series to encourage students majoring in STEM fields to gain development experience using industry APIs (application programming interfaces) for career readiness. That first episode provided an overview of the G Suite developer landscape while this episode dives deeper, introducing G Suite's HTTP-based RESTful APIs, starting with Google Drive.
The first code sample has a corresponding codelab (a self-paced, hands-on tutorial) where you can build a simple Python script that displays the first 100 files or folders in your Google Drive. The codelab helps student (and professional) developers...
Posted by Andrew Zaldivar, Developer Advocate, Google AI
A few months ago, we announced our AI Principles, a set of commitments we are upholding to guide our work in artificial intelligence (AI) going forward. Along with our AI Principles, we shared a set of recommended practices to help the larger community design and build responsible AI systems.
In particular, one of our AI Principles speaks to the importance of recognizing that AI algorithms and datasets are the product of the environment—and, as such, we need to be conscious of any potential unfair outcomes generated by an AI system and the risk it poses across cultures and societies. A recommended practice here for practitioners is to understand the limitations of their algorithm and datasets—but this is a problem that is far from solved.
To help practitioners take on the challenge of building fairer and more inclusive AI systems, we developed a short, self-study training module on fairness in machine learning. This new module is part of our Machine Learning Crash Course, which we highly recommend taking first—unless you know machine learning really well, in which case you can jump right into the Fairness module.
The Fairness module features a hands-on technical exercise. This exercise demonstrates how you can use tools and techniques that may already exist in your development stack (such as Facets Dive, Seaborn, pandas, scikit-learn and TensorFlow Estimators to name a few) to explore and discover ways to make your machine learning system fairer and more inclusive. We created our exercise in a Colaboratory notebook, which you are more than welcome to use, modify and distribute for your own purposes.
From exploring datasets to analyzing model performance, it's really easy to forget to make time for responsible reflection when building an AI system. So rather than having you run every code cell in sequential order without pause, we added what we call FairAware tasks throughout the exercise. FairAware tasks help you zoom in and out of the problem space. That way, you can remind yourself of the big picture: finding the undesirable biases that could disproportionately affect model performance across groups. We hope a process like FairAware will become part of your workflow, helping you find opportunities for inclusion.
FairAware task guiding practitioner to compare performances across gender.
The Fairness module was created to provide you with enough of an understanding to get started in addressing fairness and inclusion in AI. Keep an eye on this space for future work as this is only the beginning.
If you wish to learn more from our other examples, check out the Fairness section of our Responsible AI Practices guide. There, you will find a full set of Google recommendations and resources. From our latest research proposal on reporting model performance with fairness and inclusion considerations, to our recently launched diagnostic tool that lets anyone investigate trained models for fairness, our resource guide highlights many areas of research and development in fairness.
Let us know what your thoughts are on our Fairness module. If you have any specific comments on the notebook exercise itself, then feel free to leave a comment on our GitHub repo.
On behalf of many contributors and supporters,
Andrew Zaldivar – Developer Advocate, Google AI
Posted by Wesley Chun (@wescpy), Developer Advocate, Google Cloud
Google Cloud Next '18 is only a few days away, and this year, there are over 500 sessions covering all aspects of cloud computing, from G Suite to the Google Cloud Platform. This is your chance to learn first-hand how to build custom solutions in G Suite alongside other developers from Independent Software Vendors (ISVs), systems integrators (SIs), and industry enterprises.
G Suite's intelligent productivity apps are secure, smart, and simple to use, so why not integrate your apps with them? If you're planning to attend the event and are wondering which sessions you should check out, here are some sessions to consider:
I look forward to meeting you in person at Next '18. In the meantime, check out the entire session schedule to find out everything it has to offer. Don't forget to swing by our "Meet the Experts" office hours (Tue-Thu), G Suite "Collaboration & Productivity" showcase demos (Tue-Thu), the G Suite Birds-of-a-Feather meetup (Wed), and the Google Apps Script & G Suite Add-ons meetup (just after the BoF on Wed). I'm excited at how we can use "all the tech" to change the world. See you soon!
Posted by Wesley Chun (@wescpy), Developer Advocate, G Suite
While most chatbots respond to user requests in a synchronous way, there are scenarios when bots don't perform actions based on an explicit user request, such as for alerts or notifications. In today's DevByte video, I'm going to show you how to send messages asynchronously to rooms or direct messages (DMs) in Hangouts Chat, the team collaboration and communication tool in G Suite.
What comes to mind when you think of a bot in a chat room? Perhaps a user wants the last quarter's European sales numbers, or maybe, they want to look up local weather or the next movie showtime. Assuming there's a bot for whatever the request is, a user will either send a direct message (DM) to that bot or @mention the bot from within a chat room. The bot then fields the request (sent to it by the Hangouts Chat service), performs any necessary magic, and responds back to the user in that "space," the generic nomenclature for a room or DM.
Our previous DevByte video for the Hangouts Chat bot framework shows developers what bots and the framework are all about as well as how to build one of these types of bots, in both Python and JavaScript. However, recognize that these bots are responding synchronously to a user request. This doesn't suffice when users want to be notified when a long-running background job has completed, when a late bus or train will be arriving soon, or when one of their servers has just gone down. Recognize that such alerts can come from a bot but also perhaps a monitoring application. In the latest episode of the G Suite Dev Show, learn how to integrate this functionality in either type of application.
From the video, you can see that alerts and notifications are "out-of-band" messages, meaning they can come in at any time. The Hangouts Chat bot framework provides several ways to send asynchronous messages to a room or DM, generically referred to as a "space." The first is the HTTP-based REST API. The other way is using what are known as "incoming webhooks."
The REST API is used by bots to send messages into a space. Since a bot will never be a human user, a Google service account is required. Once you create a service account for your Hangouts Chat bot in the developers console, you can download its credentials needed to communicate with the API. Below is a short Python sample snippet that uses the API to send a message asynchronously to a space.
from apiclient import discovery from httplib2 import Http from oauth2client.service_account import ServiceAccountCredentials SCOPES = 'https://www.googleapis.com/auth/chat.bot' creds = ServiceAccountCredentials.from_json_keyfile_name( 'svc_acct.json', SCOPES) CHAT = discovery.build('chat', 'v1', http=creds.authorize(Http())) room = 'spaces/<ROOM-or-DM>' message = {'text': 'Hello world!'} CHAT.spaces().messages().create(parent=room, body=message).execute()
The alternative to using the API with service accounts is the concept of incoming webhooks. Webhooks are a quick and easy way to send messages into any room or DM without configuring a full bot, i.e., monitoring apps. Webhooks also allow you to integrate your custom workflows, such as when a new customer is added to the corporate CRM (customer relationship management system), as well as others mentioned above. Below is a Python snippet that uses an incoming webhook to communicate into a space asynchronously.
import requests import json URL = 'https://chat.googleapis.com/...&thread;_key=T12345' message = {'text': 'Hello world!'} requests.post(URL, data=json.dumps(message))
Since incoming webhooks are merely endpoints you HTTP POST to, you can even use curl to send a message to a Hangouts Chat space from the command-line:
curl
curl \ -X POST \ -H 'Content-Type: application/json' \ 'https://chat.googleapis.com/...&thread;_key=T12345' \ -d '{"text": "Hello!"}'
To get started, take a look at the Hangouts Chat developer documentation, especially the specific pages linked to above. We hope this video helps you take your bot development skills to the next level by showing you how to send messages to the Hangouts Chat service asynchronously.
We recently introduced Hangouts Chat to general availability. This next-generation messaging platform gives G Suite users a new place to communicate and to collaborate in teams. It features archive & search, tighter G Suite integration, and the ability to create separate, threaded chat rooms. The key new feature for developers is a bot framework and API. Whether it's to automate common tasks, query for information, or perform other heavy-lifting, bots can really transform the way we work.
In addition to plain text replies, Hangouts Chat can also display bot responses with richer user interfaces (UIs) called cards which can render header information, structured data, images, links, buttons, etc. Furthermore, users can interact with these components, potentially updating the displayed information. In this latest episode of the G Suite Dev Show, developers learn how to create a bot that features an updating interactive card.
As you can see in the video, the most important thing when bots receive a message is to determine the event type and take the appropriate action. For example, a bot will perform any desired "paperwork" when it is added to or removed from a room or direct message (DM), generically referred to as a "space" in the vernacular.
Receiving an ordinary message sent by users is the most likely scenario; most bots do "their thing" here in serving the request. The last event type occurs when a user clicks on an interactive card. Similar to receiving a standard message, a bot performs its requisite work, including possibly updating the card itself. Below is some pseudocode summarizing these four event types and represents what a bot would likely do depending on the event type:
function processEvent(req, rsp) { var event = req.body; // event type received var message; // JSON response message if (event.type == 'REMOVED_FROM_SPACE') { // no response as bot removed from room return; } else if (event.type == 'ADDED_TO_SPACE') { // bot added to room; send welcome message message = {text: 'Thanks for adding me!'}; } else if (event.type == 'MESSAGE') { // message received during normal operation message = responseForMsg(event.message.text); } else if (event.type == 'CARD_CLICKED') { // user-click on card UI var action = event.action; message = responseForClick( action.actionMethodName, action.parameters); } rsp.send(message); };
The bot pseudocode as well as the bot featured in the video respond synchronously. Bots performing more time-consuming operations or those issuing out-of-band notifications, can send messages to spaces in an asynchronous way. This includes messages such as job-completed notifications, alerts if a server goes down, and pings to the Sales team when a new lead is added to the CRM (Customer Relationship Management) system.
Hangouts Chat supports more than JavaScript or Python and Google Apps Script or Google App Engine. While using JavaScript running on Apps Script is one of the quickest and simplest ways to get a bot online within your organization, it can easily be ported to Node.js for a wider variety of hosting options. Similarly, App Engine allows for more scalability and supports additional languages (Java, PHP, Go, and more) beyond Python. The bot can also be ported to Flask for more hosting options. One key takeaway is the flexibility of the platform: developers can use any language, any stack, or any cloud to create and host their bot implementations. Bots only need to be able to accept HTTP POST requests coming from the Hangouts Chat service to function.
At Google I/O 2018 last week, the Hangouts Chat team leads and I delivered a longer, higher-level overview of the bot framework. This comprehensive tour of the framework includes numerous live demos of sample bots as well as in a variety of languages and platforms. Check out our ~40-minute session below.
To help you get started, check out the bot framework launch post. Also take a look at this post for a deeper dive into the Python App Engine version of the vote bot featured in the video. To learn more about developing bots for Hangouts Chat, review the concepts guides as well as the "how to" for creating bots. You can build bots for your organization, your customers, or for the world. We look forward to all the exciting bots you're going to build!
Posted by Wesley Chun (@wescpy), Developer Advocate, Google Apps
At Google I/O 2016, we launched a new Google Sheets API—click here to watch the entire announcement. The updated API includes many new features that weren’t available in previous versions, including access to functionality found in the Sheets desktop and mobile user interfaces. My latest DevByte video shows developers how to get data into and out of a Google Sheet programmatically, walking through a simple script that reads rows out of a relational database and transferring the data to a brand new Google Sheet.
Let’s take a sneak peek of the code covered in the video. Assuming that SHEETS has been established as the API service endpoint, SHEET_ID is the ID of the Sheet to write to, and data is an array with all the database rows, this is the only call developers need to make to write that raw data into the Sheet:
SHEETS
SHEET_ID
data
SHEETS.spreadsheets().values().update(spreadsheetId=SHEET_ID, range='A1', body=data, valueInputOption='RAW').execute()
rows = SHEETS.spreadsheets().values().get(spreadsheetId=SHEET_ID, range='Sheet1').execute().get('values', []) for row in rows: print(row)
If you’re ready to get started, take a look at the Python or other quickstarts in a variety of languages before checking out the DevByte. If you want a deeper dive into the code covered in the video, check out the post at my Python blog. Once you get going with the API, one of the challenges developers face is in constructing the JSON payload to send in API calls—the common operations samples can really help you with this. Finally, if you’re ready to get going with a meatier example, check out our JavaScript codelab where you’ll write a sample Node.js app that manages customer orders for a toy company, the database of which is used in this DevByte, preparing you for the codelab.
We hope all these resources help developers create amazing applications and awesome tools with the new Google Sheets API! Please subscribe to our channel, give us your feedback below, and tell us what topics you would like to see in future episodes!
Posted by Josh Gordon, Developer Advocate
To help you get started building applications with machine learning, we’re excited to launch a new developer show, Machine Learning: Recipes for New Developers. In the first few episodes, we’ll teach you the ropes of machine learning without requiring any major prerequisites (like calculus). As the series progresses, we’ll walk you from “Hello World” to solving some real world problems.
Episodes will generally publish bi-weekly, and be only about 5-10 minutes in length to keep the material lightweight. Occasionally, we’ll have guests on the show who work with machine learning on different teams around Google.
Ep #1: Hello World.
Also: Coffee with a Googler came to NYC! Laurence and Josh talk about the importance of machine learning for developers, and reducing barriers to machine learning education. Check out the video!
people
userID
import gdata.analytics.clientAPP_NAME = 'goal_names_demo'my_client = gdata.analytics.client.AnalyticsClient(source=APP_NAME)# Authorizemy_client.client_login( INSERT_USER_NAME, INSERT_PASSWORD, APP_NAME, service='analytics')# Make a query.query = gdata.analytics.client.GoalQuery( acct_id='INSERT_ACCOUNT_ID', web_prop_id='INSERT_WEB_PROP_ID', profile_id='INSERT_PROFILE_ID')# Get and print results.results = my_client.GetManagementFeed(query)for entry in results.entry: print 'Goal number = %s' % entry.goal.number print 'Goal name = %s' % entry.goal.name print 'Goal value = %s' % entry.goal.value
I've always hoped that I could release Mondrian as open source, but it was not to be: due to its popularity inside Google, it became more and more tied to proprietary Google infrastructure like Bigtable, and it remained limited to Perforce, the commercial revision control system most used at Google.What I'm announcing now is the next best thing: an code review tool for use with Subversion, inspired by Mondrian and (soon to be) released as open source. Some of the code is even directly derived from Mondrian. Most of the code is new though, written using Django and running on Google App Engine.I'm inviting the Python developer community to try out the tool on the web for code reviews. I've added a few code reviews already, but I'm hoping that more developers will upload at least one patch for review and invite a reviewer to try it out.
