tencent cloud

This document provides a detailed description that makes it easy for Native project developers to debug and integrate the APIs for Game Multimedia Engine (GME).

This document only provides the main APIs to help you get started with GME to debug and integrate the APIs.

Key Considerations for Using GME

GME provides two services: Voice chat service and voice messaging and speech-to-text service, both of which rely on key APIs such as Init and Poll.

Note on Init API

If you need to use voice chat and voice messaging services at the same time, you only need to call Init API once.

API call flowchart

Directions

Core APIs

Initializing GMEAPI: Init
Calling Poll periodically to trigger event callbacksAPI: Poll

Voice Chat

Voice Message

• Uninitializing GMEAPI: UnInit

Core API Integration

1. Download the SDK

On the SDK download guide page, download the appropriate client SDKDownLoad.

2. Importing the header file

import com.tencent.TMG.ITMGContext;
import com.tencent.av.sig.AuthBuffer;
import com.tencent.bugly.crashreport.CrashReport;

#import "GMESDK/TMGEngine.h"
#import "GMESDK/QAVAuthBuffer.h"

#include "auth_buffer.h"
#include "tmg_sdk.h"
#include "AdvanceHeaders/tmg_sdk_adv.h"
#include <vector>

3. Getting singleton

To use the voice feature, get the ITMGContext object first.

Function prototype

public static ITMGContext GetInstance(Context context)

+ (ITMGContext*) GetInstance;

__UNUSED static ITMGContext* ITMGContextGetInstance(){
return ITMGContextGetInstanceInner(TMG_SDK_VERSION);
}

Sample code

//MainActivity.java
import com.tencent.TMG.ITMGContext; 
ITMGContext tmgContext = ITMGContext.GetInstance(this);

//TMGSampleViewController.m
ITMGContext* _context = [ITMGContext GetInstance];

ITMGContext* context = ITMGContextGetInstance();

4. Setting callback

The API class uses the Delegate method to send callback notifications to the application. Register the callback function to the SDK for receiving callback messages before room entry.

Function prototype and sample code

Register the callback function to the SDK for receiving callback messages before room entry.

//ITMGContext
public abstract int SetTMGDelegate(ITMGDelegate delegate);

//MainActivity.java
tmgContext.SetTMGDelegate(TMGCallbackDispatcher.getInstance());

ITMGDelegate < NSObject >

//TMGSampleViewController.m
ITMGContext* _context = [ITMGContext GetInstance];
_context.TMGDelegate = [DispatchCenter getInstance];

// When initializing the SDK
m_pTmgContext = ITMGContextGetInstance();
m_pTmgContext->SetTMGDelegate(this);
// In the destructor
CTMGSDK_For_AudioDlg::~CTMGSDK_For_AudioDlg()
{
    ITMGContextGetInstance()->SetTMGDelegate(NULL);
}

Callback examples

Override this callback function in the constructor to process the parameters of the callback.

//MainActivity.java
tmgContext.SetTMGDelegate(TMGCallbackDispatcher.getInstance());

//RealTimeVoiceActivity.java
public void OnEvent(ITMGContext.ITMG_MAIN_EVENT_TYPE type, Intent data) {
        if (type == ITMG_MAIN_EVENT_TYPE_ENTER_ROOM)
        {
            // Processing callbacks
        }
}

// Refer to TMGCallbackDispatcher.java, TMGCallbackHelper.java, and TMGDispatcherBase.java

//TMGRealTimeViewController.m
TMGRealTimeViewController ()< ITMGDelegate >


- (void)OnEvent:(ITMG_MAIN_EVENT_TYPE)eventType data:(NSDictionary *)data {
  NSString *log = [NSString stringWithFormat:@"OnEvent:%d,data:%@", (int)eventType, data];
  [self showLog:log];
  NSLog(@"====%@====", log);
  switch (eventType) {
      // Step 6/11 : Perform the enter room event
      case ITMG_MAIN_EVENT_TYPE_ENTER_ROOM: {
          int result = ((NSNumber *)[data objectForKey:@"result"]).intValue;
          NSString *error_info = [data objectForKey:@"error_info"];

          [self showLog:[NSString stringWithFormat:@"OnEnterRoomComplete:%d msg:(%@)", result, error_info]];
      
          if (result == 0) {
              [self updateStatusEnterRoom:YES];
          }
      }
      break;

  }
  }

// Refer to DispatchCenter.h and DispatchCenter.m

// Declaration in the header file
virtual void OnEvent(ITMG_MAIN_EVENT_TYPE eventType,const char* data);
// Sample code
void CTMGSDK_For_AudioDlg::OnEvent(ITMG_MAIN_EVENT_TYPE eventType, const char* data)
{
    switch(eventType)
    {
    case ITMG_MAIN_EVENT_TYPE_XXXX_XXXX:
        {
            // Process the callback
        }
        break;
    }
}

5. Initializing SDK

You need to initialize the SDK through the Init API before you can use the real-time voice, voice message, and speech-to-text services. The Init API must be called in the same thread as other APIs. We recommend you call all APIs in the main thread.

API prototype

public abstract int Init(String sdkAppId, String openId);

-(int)InitEngine:(NSString*)sdkAppID openID:(NSString*)openID;

ITMGContext virtual int Init(const char* sdkAppId, const char* openId)

Sample code

//MainActivity.java
int nRet = tmgContext.Init(appId, openId);
if (nRet == AV_OK )
{
    GMEAuthBufferHelper.getInstance().setGEMParams(appId, key, openId);
    // Step 4/11: Poll to trigger callback
    //https://www.tencentcloud.com/document/product/607/40860
    EnginePollHelper.createEnginePollHelper();
    showToast("Init success");
}else if (nRet == AV_ERR_HAS_IN_THE_STATE) // SDK has been initialized. This operation is successful.
{
    showToast("Init success");
}else
{
    showToast("Init error errorCode:" + nRet);
}

//TMGSampleViewController.m
QAVResult result = [_context InitEngine:self.appIDTF.text openID:self.openIDTF.text];
if (result == QAV_OK) {
    self.isSDKInit = YES;
}

#define SDKAPPID3RD "14000xxxxx"
cosnt char* openId="10001";
ITMGContext* context = ITMGContextGetInstance();
context->Init(SDKAPPID3RD, openId);

6. Triggering event callback

Event callbacks can be triggered by periodically calling the Poll API in update. The Poll API is GME's message pump and should be called periodically for GME to trigger event callbacks; otherwise, the entire SDK service will run abnormally. For more information, see the EnginePollHelper file in SDK Download Guide.

Sample code

//MainActivity.java
[EnginePollHelper createEnginePollHelper];

//EnginePollHelper.java
private Handler mhandler = new Handler();
    private Runnable mRunnable = new Runnable() {
        @Override
        public void run() {
            if (s_pollEnabled) {
                if (ITMGContext.GetInstance(null) != null)
                    ITMGContext.GetInstance(null).Poll();
            }
            mhandler.postDelayed(mRunnable, 33);
        }
    };
// For the code of calling Poll periodically, see EnginePollHelper.java.

//TMGSampleViewController.m
[EnginePollHelper createEnginePollHelper];
// Refer to EnginePollHelper.m and EnginePollHelper.h

void TMGTestScene::update(float delta)
{
  ITMGContextGetInstance()->Poll();
}

7. Calculating the local authentication key

Generate AuthBuffer for encryption and authentication of relevant features. For release in the production environment, please use the backend deployment key as detailed in Authentication Key.

API prototype

AuthBuffer public native byte[] genAuthBuffer(int sdkAppId, String roomId, String openId, String key)

//TMGSampleViewController.m
[EnginePollHelper createEnginePollHelper];
// Refer to EnginePollHelper.m and EnginePollHelper.h

void TMGTestScene::update(float delta)
{
  ITMGContextGetInstance()->Poll();
}

Sample code

//GMEAuthBufferHelper.java
import com.tencent.av.sig.AuthBuffer;// Header file
public byte[] createAuthBuffer(String roomId)
    {
        byte[] authBuffer;
        // Generate AuthBuffer for encryption and authentication of relevant features. For release in the production environment,
        // please use the backend deployment key as detailed in https://www.tencentcloud.com/document/product/607/12218
        if (TextUtils.isEmpty(roomId))
        {
            authBuffer =  AuthBuffer.getInstance().genAuthBuffer(Integer.parseInt(mAppId), "0", mOpenId, mKey);
        }else
        {
            authBuffer =  AuthBuffer.getInstance().genAuthBuffer(Integer.parseInt(mAppId), roomId, mOpenId, mKey);
        }
        return authBuffer;
    }

// Voice chat authentication 
NSData* authBuffer = [QAVAuthBuffer GenAuthBuffer:SDKAPPID3RD.intValue roomID:self.roomIdTF.text openID:_openId key:_key];
// Voice message authentication
NSData* authBuffer =  [QAVAuthBuffer GenAuthBuffer:(unsigned int)SDKAPPID3RD.integerValue roomID:nil openID:self.openId key:AUTHKEY];

unsigned int bufferLen = 512;
unsigned char retAuthBuff[512] = {0};
QAVSDK_AuthBuffer_GenAuthBuffer(atoi(SDKAPPID3RD), roomId, "10001", AUTHKEY,retAuthBuff,bufferLen);

Voice Chat Access

1. Entering a room

This API is used to enter a room with the generated authentication information. The mic and speaker are not turned on by default after room entry. The returned value of AV_OK indicates successful API call but not successful room entry.

API prototype

public abstract int EnterRoom(String roomID, int roomType, byte[] authBuffer);

-(int)EnterRoom:(NSString*) roomId roomType:(int)roomType authBuffer:(NSData*)authBuffer;

ITMGContext virtual int EnterRoom(const char*  roomID, ITMG_ROOM_TYPE roomType, const char* authBuff, int buffLen);

Sample code

//RealTimeVoiceActivity.java
byte[] authBuffer =  GMEAuthBufferHelper.getInstance().createAuthBuffer(roomId);
ITMGContext.GetInstance(this).EnterRoom(roomId, roomType, authBuffer);

//TMGRealTimeViewController.m
[[ITMGContext GetInstance] EnterRoom:self.roomIdTF.text roomType:(int)self.roomTypeControl.selectedSegmentIndex + 1 authBuffer:authBuffer];

ITMGContext* context = ITMGContextGetInstance();
context->EnterRoom(roomID, ITMG_ROOM_TYPE_FLUENCY, (char*)retAuthBuff,bufferLen);

Callback for room entry

After the user enters the room, the message ITMG_MAIN_EVENT_TYPE_ENTER_ROOM will be sent and identified in the OnEvent function for callback and processing. A successful callback means that the room entry is successful, and the billing starts.

• Sample code
  Sample code for processing the callback, including room entry and network disconnection events.
  • Java Java
  • Object-C objetctive-c
  • C++ C++

  //RealTimeVoiceActivity.java
  public void OnEvent(ITMGContext.ITMG_MAIN_EVENT_TYPE type, Intent data) {
      if (type == ITMG_MAIN_EVENT_TYPE_ENTER_ROOM)
      {
          // Step 6/11 : Perform the enter room event
          int nErrCode = TMGCallbackHelper.ParseIntentParams2(data).nErrCode;
          String strMsg = TMGCallbackHelper.ParseIntentParams2(data).strErrMsg;
          if (nErrCode == AV_OK)
          {
              appendLog2MonitorView("EnterRomm success");
          }else
          {
              appendLog2MonitorView(String.format(Locale.getDefault(), "EnterRomm errCode:%d errMsg:%s", nErrCode, strMsg));
          }
      }
  }

  //TMGRealTimeViewController.m
  
  - (void)OnEvent:(ITMG_MAIN_EVENT_TYPE)eventType data:(NSDictionary *)data {
    NSString *log = [NSString stringWithFormat:@"OnEvent:%d,data:%@", (int)eventType, data];
    [self showLog:log];
    NSLog(@"====%@====", log);
    switch (eventType) {
        // Step 6/11 : Perform the enter room event
        case ITMG_MAIN_EVENT_TYPE_ENTER_ROOM: {
            int result = ((NSNumber *)[data objectForKey:@"result"]).intValue;
            NSString *error_info = [data objectForKey:@"error_info"];
  
            [self showLog:[NSString stringWithFormat:@"OnEnterRoomComplete:%d msg:(%@)", result, error_info]];
        
            if (result == 0) {
                [self updateStatusEnterRoom:YES];
            }
        }
        break;
  
    }

  void TMGTestScene::OnEvent(ITMG_MAIN_EVENT_TYPE eventType,const char* data){
  switch (eventType) {
      case ITMG_MAIN_EVENT_TYPE_ENTER_ROOM:
      {
          ListMicDevices();
          ListSpeakerDevices();
              std::string strText = "EnterRoom complete: ret=";
          strText += data;
          m_EditMonitor.SetWindowText(MByteToWChar(strText).c_str());
          }
  }
  }

• Error code

  Error Code Value	Cause and Suggested Solution
  7006	Authentication failed. Possible causes: The `AppID` does not exist or is incorrect. An error occurred while authenticating the `authbuff`. Authentication expired. The `openId` does not meet the specification.
  7007	Already in another room.
  1001	The user was already in the process of entering a room but repeated this operation. It is recommended not to call the room entering API until the room entry callback is returned.
  1003	The user was already in the room and called the room entering API again.
  1101	Make sure that the SDK is initialized, `openId` complies with the rules, the APIs are called in the same thread, and the `Poll` API is called normally.

2. Turning on or off the microphone

This API is used to turn on or off the mic. Mic and speaker are not enabled by default after room entry.

Sample code

//RealTimeVoiceActivity.java
ITMGContext.GetInstance(this).GetAudioCtrl().EnableMic(true);

//TMGRealTimeViewController.m
[[[ITMGContext GetInstance] GetAudioCtrl] EnableMic:YES];

ITMGContextGetInstance()->GetAudioCtrl()->EnableMic(true);

3. Turning on or off the speaker

This API is used to turn on/off the speaker.

Sample code

//RealTimeVoiceActivity.java
ITMGContext.GetInstance(this).GetAudioCtrl().EnableSpeaker(true);

//TMGRealTimeViewController.m
[[[ITMGContext GetInstance] GetAudioCtrl] EnableSpeaker:YES];

ITMGContextGetInstance()->GetAudioCtrl()->EnableSpeaker(true);

4. Exiting the room

This API is called to exit the current room. It needs to wait for and process the callback for exit.

Sample code

//RealTimeVoiceActivity.java
ITMGContext.GetInstance(this).ExitRoom();

//TMGRealTimeViewController.m
[[ITMGContext GetInstance] ExitRoom];

ITMGContext* context = ITMGContextGetInstance();
context->ExitRoom();

Callback for room exit

After the user exits a room, a callback will be returned with the message being ITMG_MAIN_EVENT_TYPE_EXIT_ROOM. The sample code is shown below:

//RealTimeVoiceActivity.java
public void OnEvent(ITMGContext.ITMG_MAIN_EVENT_TYPE type, Intent data) {
    if (ITMGContext.ITMG_MAIN_EVENT_TYPE.ITMG_MAIN_EVENT_TYPE_EXIT_ROOM == type)
        {
            // Receive the event of successful room exit
        }
}

//TMGRealTimeViewController.m
-(void)OnEvent:(ITMG_MAIN_EVENT_TYPE)eventType data:(NSDictionary *)data{
NSLog(@"OnEvent:%lu,data:%@",(unsigned long)eventType,data);
switch (eventType) {
    case ITMG_MAIN_EVENT_TYPE_EXIT_ROOM:
    {
        // Receive the event of successful room exit
    }
        break;
    }
}

void TMGTestScene::OnEvent(ITMG_MAIN_EVENT_TYPE eventType,const char* data){
switch (eventType) {
    case ITMG_MAIN_EVENT_TYPE_EXIT_ROOM:
    {
        // Process
        break;
        }
    }
}

Voice Message Access

1. Initializing authentication

Call authentication initialization after initializing the SDK. For more information on how to get the authBuffer, please see genAuthBuffer (the voice chat authentication information API).

API prototype

public abstract int ApplyPTTAuthbuffer(byte[] authBuffer);

-(QAVResult)ApplyPTTAuthbuffer:(NSData *)authBuffer;

ITMGPTT virtual int ApplyPTTAuthbuffer(const char* authBuffer, int authBufferLen)

Sample code

//VoiceMessageRecognitionActivity.java
byte[] authBuffer = GMEAuthBufferHelper.getInstance().createAuthBuffer("");
ITMGContext.GetInstance(this).GetPTT().ApplyPTTAuthbuffer(authBuffer);

//TMGPTTViewController.m
NSData* authBuffer =  [QAVAuthBuffer GenAuthBuffer:(unsigned int)SDKAPPID3RD.integerValue roomID:nil openID:self.openId key:AUTHKEY];
[[[ITMGContext GetInstance] GetPTT] ApplyPTTAuthbuffer:authBuffer];

ITMGContextGetInstance()->GetPTT()->ApplyPTTAuthbuffer(authBuffer,authBufferLen);

2. Starting streaming speech recognition

This API is used to start streaming speech recognition. Text obtained from speech-to-text conversion will be returned in real time in its callback. To stop recording, call StopRecording. The callback will be returned after the recording is stopped.

API prototype

public abstract int StartRecordingWithStreamingRecognition (String filePath);

public abstract int StopRecording();

-(int)StartRecordingWithStreamingRecognition:(NSString *)filePath;

-(QAVResult)StopRecording;

ITMGPTT virtual int StartRecordingWithStreamingRecognition(const char* filePath) 

ITMGPTT virtual int StopRecording()

Sample code

//VoiceMessageRecognitionActivity.java
ITMGContext.GetInstance(this).GetPTT().StartRecordingWithStreamingRecognition(recordfilePath);

//TMGPTTViewController.m
QAVResult ret = [[[ITMGContext GetInstance] GetPTT] StartRecordingWithStreamingRecognition:[self pttTestPath]];
if (ret == 0) {
    self.currentStatus = @"Start streaming recording";
} else {
    self.currentStatus = @"Failed to start streaming recording";
}

ITMGContextGetInstance()->GetPTT()->StartRecordingWithStreamingRecognition(filePath);

Callback for streaming speech recognition

After streaming speech recognition is started, you need to listen for callback messages in the callback function onEvent. The event message is ITMG_MAIN_EVNET_TYPE_PTT_STREAMINGRECOGNITION_COMPLETE, namely returns text after the recording is stopped and the recognition is completed, which is equivalent to returning the recognized text after a paragraph of speech.

The event message will be identified in the OnEvent function based on the actual needs. The passed parameters include the following four messages.

• Sample code
  • Java Java
  • Object-C objetctive-c
  • C++ C++

  //VoiceMessageRecognitionActivity.java
  import static com.tencent.TMG.ITMGContext.ITMG_MAIN_EVENT_TYPE.ITMG_MAIN_EVNET_TYPE_PTT_STREAMINGRECOGNITION_COMPLETE;
  public void OnEvent(ITMGContext.ITMG_MAIN_EVENT_TYPE type, Intent data) {
      if (type == ITMG_MAIN_EVNET_TYPE_PTT_STREAMINGRECOGNITION_COMPLETE)
          {
              // Step 1.3/3 handle ITMG_MAIN_EVNET_TYPE_PTT_STREAMINGRECOGNITION_COMPLETE event
              mIsRecording = false;
              if (nErrCode ==0)
              {
                  String recordfilePath = data.getStringExtra("file_path");
                  mRecFilePathView.setText(recordfilePath);
  
                  String recordFileUrl = data.getStringExtra("file_id");
                  mRecFileUrlView.setText(recordFileUrl);
              }
              else
              {
                  appendLog2MonitorView("Record and recognition fail errCode:" + nErrCode);
              }
          }
  
  }

  //TMGPTTViewController.m
  
  - (void)OnEvent:(ITMG_MAIN_EVENT_TYPE)eventType data:(NSDictionary*)data
    {
    NSNumber *number = [data objectForKey:@"result"];
    switch (eventType)
    {
        case ITMG_MAIN_EVNET_TYPE_PTT_STREAMINGRECOGNITION_COMPLETE:
        {
              if (data != NULL &&[[data objectForKey:@"result"] intValue]== 0)
              {
                  self.translateTF.text = [data objectForKey:@"text"] ;
                   self.currentStatus = @"Streaming conversion completed";
              }
          }
        break;
    }

  void TMGTestScene::OnEvent(ITMG_MAIN_EVENT_TYPE eventType,const char* data){
    switch (eventType) {
        case ITMG_MAIN_EVNET_TYPE_PTT_STREAMINGRECOGNITION_COMPLETE:
        {
            HandleSTREAM2TEXTComplete(data,true);
            break;
        }
        ...
        case ITMG_MAIN_EVNET_TYPE_PTT_STREAMINGRECOGNITION_IS_RUNNING:
        {
            HandleSTREAM2TEXTComplete(data, false);
            break;
        }
    }
  }
  void CTMGSDK_For_AudioDlg::HandleSTREAM2TEXTComplete(const char* data, bool isComplete)
  {
    std::string strText = "STREAM2TEXT: ret=";
    strText += data;
    m_EditMonitor.SetWindowText(MByteToWChar(strText).c_str());
    Json::Reader reader;
    Json::Value root;
    bool parseRet = reader.parse(data, root);
    if (!parseRet) {
        ::SetWindowText(m_EditInfo.GetSafeHwnd(),MByteToWChar(std::string("parse result Json error")).c_str());
    }
        else
        {
            if (isComplete) {
                                ::SetWindowText(m_EditUpload.GetSafeHwnd(), MByteToWChar(root["file_id"].asString()).c_str());
                            }
                            else {
                                    std::string isruning = "STREAMINGRECOGNITION_IS_RUNNING";
                                    ::SetWindowText(m_EditUpload.GetSafeHwnd(), MByteToWChar(isruning).c_str());
                                 }
    }
  }

• Error code

  Error Code	Description	Suggested Solution
  32775	Streaming speech-to-text conversion failed, but recording succeeded.	Call the `UploadRecordedFile` API to upload the recording file and then call the `SpeechToText` API to perform speech-to-text conversion.
  32777	Streaming speech-to-text converting failed, but recording and upload succeeded	The message returned contains a backend URL after successful upload. Call the `SpeechToText` API to perform speech-to-text conversion.
  32786	Streaming speech-to-text conversion failed.	During streaming recording, wait for the execution result of the streaming recording API to return.

3. Stopping recording

This API is used to stop recording. It is async, and a callback for recording completion will be returned after recording stops. A recording file will be available only after recording succeeds.

API prototype

public abstract int StopRecording();

-(QAVResult)StopRecording;

ITMGPTT virtual int StopRecording();

Sample code

//VoiceMessageRecognitionActivity.java
ITMGContext.GetInstance(this).GetPTT().StopRecording();

//TMGPTTViewController.m

- (void)stopRecClick {
  // Step 3/12 stop recording,  need handle ITMG_MAIN_EVNET_TYPE_PTT_RECORD_COMPLETE event
  // https://www.tencentcloud.com/document/product/607/15221
  QAVResult ret =  [[[ITMGContext GetInstance] GetPTT] StopRecording];
   if (ret == 0) {
       self.currentStatus = @"Stop recording";
   } else {
        self.currentStatus = @"Failed to stop recording";
   }
  }

ITMGContextGetInstance()->GetPTT()->StopRecording();