MelodyMap

Project Overview

MelodyMap is an innovative Android application that bridges the gap between visual experiences and music by automatically generating context-aware Spotify playlists from video scenes. The app uses advanced video processing and Google's Gemini Vision AI to analyze scenes and create perfectly matched musical accompaniments.

Key Features

• Video Capture & Processing: Real-time video recording with frame extraction
• Scene Analysis: Advanced scene understanding using Gemini Vision AI
• Playlist Generation: Automated Spotify playlist creation based on scene context
• Seamless Integration: Direct playlist access through Spotify's platform

Technical Implementation

Video Processing System

The app implements video processing capabilities:

class VideoProcesser {
    fun processVideo(context: Context, videoFilePath: String, callback: (List<Bitmap>) -> Unit) {
        val frames = mutableListOf<Bitmap>()
        val retriever = MediaMetadataRetriever()
        
        // Extract key frames for analysis
        val duration = retriever.extractMetadata(MediaMetadataRetriever.METADATA_KEY_DURATION)?.toLong() ?: 0L
        val interval = duration / 8
        
        // Process frames at regular intervals
        for (i in 0 until 8) {
            val frameTime = i * interval * 1000
            val bitmap = retriever.getFrameAtTime(frameTime)
            bitmap?.let { frames.add(it) }
        }
    }
}

AI Integration

The application leverages Google's Gemini Vision API for advanced scene analysis:

class GeminiProVision {
    fun getResponse(query: String, frames: List<Bitmap>, callback: ResponseCallback) {
        // Configure AI model parameters
        val configBuilder = GenerationConfig.Builder().apply {
            temperature = 0.9f
            topK = 16
            topP = 0.1f
        }
        
        // Process frames through Gemini Vision
        val content = Content.Builder()
            .addText(query)
            .apply { frames.forEach { addImage(it) } }
            .build()
            
        // Generate music recommendations
        model.generateContent(content)
    }
}

Spotify Integration

Seamless integration with Spotify's API for playlist management:

class SpotifyHelper(private val context: Context) {
    fun createPlaylist(accessToken: String, userId: String, playlistName: String): String {
        // Create new playlist
        val json = JSONObject().apply {
            put("name", playlistName)
            put("description", "Generated by MelodyMap")
            put("public", false)
        }
        
        // Add tracks based on AI recommendations
        return spotifyApiService.createPlaylist(userId, json).execute()
            .body()?.id ?: throw Exception("Failed to create playlist")
    }
}

Technical Architecture

The application follows a clean architecture pattern with distinct layers:

1. Presentation Layer

   • Activity-based UI components
   • Real-time video preview
   • Playlist interaction interface

2. Domain Layer

   • Video processing logic
   • Frame extraction and analysis
   • Playlist generation algorithms

3. Data Layer

   • Spotify API integration
   • Gemini Vision API communication
   • Local data persistence

Technical Stack

• Platform: Android (API Level 24+)
• Language: Kotlin
• APIs:
  • Spotify Web API
  • Google Gemini Vision API
• Libraries:
  • Android Camera2 API
  • Retrofit for networking
  • MediaRecorder for video capture
  • Android Navigation Components

Results and Impact

• Seamless integration of video analysis and music recommendation
• Real-time video processing with efficient frame extraction
• Context-aware playlist generation using advanced AI
• Direct Spotify playlist creation and management

Future Development

• Enhanced scene recognition capabilities
• Multi-platform support (iOS)
• Expanded music service integrations
• Social sharing features
• Custom playlist customization options