Yes, there is a hook provided in the theme for this : badgeos_wplms_start_course

1. Go to WP Admin -> Plugins -> editor -> WPLMS Customizer -> customizer_class.php

2. Add following code in __construct function :

PHP Code:

 

add_action('badgeos_wplms_start_course',array($this,'send_notification_to_instructor'),10,1);

 

3. Add following function in class:

PHP Code:

 

function send_notification_to_instructor($course_id){

$instructors=apply_filters('wplms_course_instructors',get_post_field('post_author',$course_id));
$user_id= get_current_user_id();

$message = 'Student '.bp_core_get_user_link($user_id).' started course '.get_the_title($course_id);
messages_new_message(array('sender_id'=>$user_id,'subject'=>'Student Started Course','content'=>$message,'recipients'=>$instructors));
}